Provided with the classic Auto MPG dataset, we will predict the fuel efficiency of the late-1970s and early 1980s automobiles, leveraging features such as cylinders, displacement, horsepower, weight, etc.
It is a very small dataset and there are only a few features. We will first build a linear model and a neural network, evaluate their performances, track our experiment runs and inspect the logs using MLflow, and apply TPOT to see how it can be used to search over many ML model architectures, followed by explaining the model with SHAP.
By the end of this session, you will be able to
Note: State of Data Science and Machine Learning 2021 by Kaggle shows that the most commonly used algorithms were linear and logtistic regressions, followed closely by decision trees, random forests, and gradient boosting machines (are you surprised?). Multilayer perceptron, or artificial neural networks are not yet the popular tools for tabular/structured data; see more technical reasons in papers: Deep Neural Networks and Tabular Data: A Survey, Tabular Data: Deep Learning is Not All You Need. For this assignment, the main purpose is for you to get familiar with the basic building blocks in constructing neural networks before we dive into more specialized neural network architectures.
IMPORTANT
You only need to run the following cells if you're completing the assignment in Google Collab. If you've already installed these libraries locally, you can skip installing these libraries.
# this notebook run local - I did the prework provided in github
# Connect colab to your Google Drive
#from google.colab import drive #coomented out
#drive.mount('/content/drive') #out
!pip install daal==2021.4.0 #needed to do the sns
Collecting daal==2021.4.0
Using cached daal-2021.4.0-py2.py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.whl (189.9 MB)
Collecting tbb==2021.*
Using cached tbb-2021.8.0-py2.py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.whl (1.0 MB)
Installing collected packages: tbb, daal
Attempting uninstall: tbb
Found existing installation: TBB 0.2
ERROR: Cannot uninstall 'TBB'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
!pip install -q pluggy==1.0.0
!pip install -q seaborn # pairplot
!pip install -q tpot # automl
!pip install -q mlflow # tracking
!pip install -q pyngrok # workaround to run mlflow ui in colab
!pip install -q shap
## one errors - most of the stuff should run
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Make NumPy printouts easier to read.
np.set_printoptions(precision=3, suppress=True)
import tensorflow as tf
from tensorflow.keras import layers
print(tf.__version__)
2023-01-31 18:27:06.927507: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2.11.0
Start MLflow's automatic logging using library-specific autolog calls for tensorflow: logging metrics, parameters, and models without the need for explicit log statements.
We will get into more details using MLflow after completing our experiment.
import mlflow
mlflow.tensorflow.autolog() # MLflow Autologging
# typed conda install -c conda-forge mlflow at the terminal prompt
pandas:url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data'
column_names = [
'MPG', 'Cylinders', 'Displacement', 'Horsepower', 'Weight',
'Acceleration', 'Model Year', 'Origin'
]
dataset = pd.read_csv(url, names=column_names, na_values='?',
comment='\t', sep=' ', skipinitialspace=True)
dataset.tail()
| MPG | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model Year | Origin | |
|---|---|---|---|---|---|---|---|---|
| 393 | 27.0 | 4 | 140.0 | 86.0 | 2790.0 | 15.6 | 82 | 1 |
| 394 | 44.0 | 4 | 97.0 | 52.0 | 2130.0 | 24.6 | 82 | 2 |
| 395 | 32.0 | 4 | 135.0 | 84.0 | 2295.0 | 11.6 | 82 | 1 |
| 396 | 28.0 | 4 | 120.0 | 79.0 | 2625.0 | 18.6 | 82 | 1 |
| 397 | 31.0 | 4 | 119.0 | 82.0 | 2720.0 | 19.4 | 82 | 1 |
pd.DataFrame.dropna():dataset.shape
(398, 8)
dataset = dataset.dropna() # YOUR CODE HERE
dataset.shape
(392, 8)
"Origin" column is categorical, not numeric. So the next step is to one-hot encode the values in the column with pd.get_dummies.dataset['Origin'] = dataset['Origin'].replace({1: 'USA', 2: 'Europe', 3: 'Japan'})
dataset = pd.get_dummies(dataset, columns=['Origin'], prefix='', prefix_sep='')
dataset.tail()
| MPG | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model Year | Europe | Japan | USA | |
|---|---|---|---|---|---|---|---|---|---|---|
| 393 | 27.0 | 4 | 140.0 | 86.0 | 2790.0 | 15.6 | 82 | 0 | 0 | 1 |
| 394 | 44.0 | 4 | 97.0 | 52.0 | 2130.0 | 24.6 | 82 | 1 | 0 | 0 |
| 395 | 32.0 | 4 | 135.0 | 84.0 | 2295.0 | 11.6 | 82 | 0 | 0 | 1 |
| 396 | 28.0 | 4 | 120.0 | 79.0 | 2625.0 | 18.6 | 82 | 0 | 0 | 1 |
| 397 | 31.0 | 4 | 119.0 | 82.0 | 2720.0 | 19.4 | 82 | 0 | 0 | 1 |
Split the data into training and test sets. To reduce the module importing overhead, instead of sklearn.model_selection.train_test_split(), use pd.DataFrame.sample() to save 80% of the data aside to train_dataset, set the random state to be 0 for reproducibility.
Then use pd.DataFrame.drop() to obtain the test_dataset.
train_dataset = dataset.sample(frac=.8,random_state=0) # YOUR CODE HERE
test_dataset = dataset.drop(train_dataset.index)# YOUR CODE HERE
Review the pairwise relationships of a few pairs of columns from the training set.
The top row suggests that the fuel efficiency (MPG) is a function of all the other parameters. The other rows indicate they are functions of each other.
########### DEBUG THIS ##########
sns.pairplot(train_dataset[['MPG', 'Cylinders', 'Displacement', 'Weight']]); #####
Let's also check the overall statistics. Note how each feature covers a very different range:
train_dataset.describe().transpose()
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| MPG | 314.0 | 23.310510 | 7.728652 | 10.0 | 17.00 | 22.0 | 28.95 | 46.6 |
| Cylinders | 314.0 | 5.477707 | 1.699788 | 3.0 | 4.00 | 4.0 | 8.00 | 8.0 |
| Displacement | 314.0 | 195.318471 | 104.331589 | 68.0 | 105.50 | 151.0 | 265.75 | 455.0 |
| Horsepower | 314.0 | 104.869427 | 38.096214 | 46.0 | 76.25 | 94.5 | 128.00 | 225.0 |
| Weight | 314.0 | 2990.251592 | 843.898596 | 1649.0 | 2256.50 | 2822.5 | 3608.00 | 5140.0 |
| Acceleration | 314.0 | 15.559236 | 2.789230 | 8.0 | 13.80 | 15.5 | 17.20 | 24.8 |
| Model Year | 314.0 | 75.898089 | 3.675642 | 70.0 | 73.00 | 76.0 | 79.00 | 82.0 |
| Europe | 314.0 | 0.178344 | 0.383413 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| Japan | 314.0 | 0.197452 | 0.398712 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| USA | 314.0 | 0.624204 | 0.485101 | 0.0 | 0.00 | 1.0 | 1.00 | 1.0 |
train_features = train_dataset.copy() # hard copy of the dataframe - original stays as is
test_features = test_dataset.copy() # YOUR CODE HERE
train_labels = train_features.pop('MPG') # this removes the MPG variable from train_features
test_labels = test_features.pop('MPG') # and moves it in the train_labels --> elegant
It is good practice to normalize features that use different scales and ranges. Although a model might converge without feature normalization, normalization makes training much more stable.
Similar to scikit-learn, tensorflow.keras offers a list of preprocessing layers so that you can build and export models that are truly end-to-end.
tf.keras.layers.Normalization is a clean and simple way to add feature normalization into your model. The first step is to create the layer:normalizer = tf.keras.layers.Normalization() # YOUR CODE HERE
Normalization.adapt:normalizer.adapt(train_features) # adapt blank stuff
We can see the feature mean and variance are stored in the layer:
print(f'feature mean: {normalizer.mean.numpy().squeeze()}\n')
print(f'feature variance: {normalizer.variance.numpy().squeeze()}')
feature mean: [ 5.478 195.318 104.869 2990.252 15.559 75.898 0.178 0.197
0.624]
feature variance: [ 2.88 10850.413 1446.699 709896.9 7.755 13.467
0.147 0.158 0.235]
When the layer is called, it returns the input data, with each feature independently normalized:
first = np.array(train_features[:1])
with np.printoptions(precision=2, suppress=True):
print('First example:', first)
print()
print('Normalized:', normalizer(first).numpy())
First example: [[ 4. 90. 75. 2125. 14.5 74. 0. 0. 1. ]] Normalized: [[-0.87 -1.01 -0.79 -1.03 -0.38 -0.52 -0.47 -0.5 0.78]]
Before building a deep neural network model, start with linear regression using all the features.
Training a model with tf.keras typically starts by defining the model architecture. Use a tf.keras.Sequential model, which represents a sequence of steps.
There are two steps in this multivariate linear regression model:
tf.keras.layers.Normalization preprocessing layer. You have defined this earlier as normalizer.tf.keras.layers.Dense).The number of inputs can either be set by the input_shape argument, or automatically when the model is run for the first time.
linear_model = tf.keras.Sequential([
normalizer,
tf.keras.layers.Dense(units=1) # could add activation='linear')
])
linear_model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
normalization (Normalizatio (None, 9) 19
n)
dense_1 (Dense) (None, 1) 10
=================================================================
Total params: 29
Trainable params: 10
Non-trainable params: 19
_________________________________________________________________
'MPG' from all features in train_features. Run the untrained model on the first 10 data points / rows using Model.predict(). The output won't be good, but notice that it has the expected shape of (10, 1):linear_model.predict(train_features[:10])## YOUR CODE HERE
1/1 [==============================] - 0s 61ms/step
array([[ 0.109],
[ 0.373],
[-1.933],
[ 1.091],
[ 1.507],
[ 0.171],
[ 1.69 ],
[ 1.625],
[-0.446],
[ 0.566]], dtype=float32)
kernel weights (the $m$ in $y = mx + b$) have a shape of (9, 1):linear_model.layers[1].kernel
<tf.Variable 'dense_1/kernel:0' shape=(9, 1) dtype=float32, numpy=
array([[-0.429],
[ 0.704],
[-0.504],
[-0.52 ],
[ 0.124],
[ 0.327],
[ 0.53 ],
[ 0.432],
[ 0.251]], dtype=float32)>
Once the model is built, configure the training procedure using the Keras Model.compile method. The most important arguments to compile are the loss and the optimizer, since these define what will be optimized and how (using the tf.keras.optimizers.Adam).
Here's a list of built-in loss functions in tf.keras.losses. For regression tasks, common loss functions include mean squared error (MSE) and mean absolute error (MAE). Here, MAE is preferred such that the model is more robust against outliers.
For optimizers, gradient descent (check this video Gradient Descent, Step-by-Step for a refresher) is the preferred way to optimize neural networks and many other machine learning algorithms. Read an overview of graident descent optimizer algorithms for several popular gradient descent algorithms. Here, we use the popular tf.keras.optimizers.Adam, and set the learning rate at 0.1 for faster learning.
linear_model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate = .1), # YOUR CODE HERE
loss=tf.keras.losses.MeanAbsoluteError() # YOUR CODE HERE
)
Model.fit to execute the training for 100 epochs, set the verbose to 0 to suppress logging and keep 20% of the data for validation:%%time
history = linear_model.fit(x=train_features,y=train_labels,epochs=100, validation_split=.2) # YOUR CODE HERE
2023/01/31 19:14:55 INFO mlflow.utils.autologging_utils: Created MLflow autologging run with ID '416c1ca789a646cba1604f5d1ec01953', which will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow
Epoch 1/100 1/8 [==>...........................] - ETA: 5s - loss: 22.7974WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0022s vs `on_train_batch_end` time: 0.0034s). Check your callbacks.
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0022s vs `on_train_batch_end` time: 0.0034s). Check your callbacks.
8/8 [==============================] - 1s 44ms/step - loss: 22.9780 - val_loss: 22.8973 Epoch 2/100 8/8 [==============================] - 0s 10ms/step - loss: 22.0896 - val_loss: 22.1690 Epoch 3/100 8/8 [==============================] - 0s 11ms/step - loss: 21.2538 - val_loss: 21.4801 Epoch 4/100 8/8 [==============================] - 0s 14ms/step - loss: 20.4357 - val_loss: 20.7823 Epoch 5/100 8/8 [==============================] - 0s 12ms/step - loss: 19.6283 - val_loss: 20.0840 Epoch 6/100 8/8 [==============================] - 0s 11ms/step - loss: 18.7895 - val_loss: 19.3546 Epoch 7/100 8/8 [==============================] - 0s 14ms/step - loss: 17.9768 - val_loss: 18.5769 Epoch 8/100 8/8 [==============================] - 0s 13ms/step - loss: 17.1567 - val_loss: 17.8816 Epoch 9/100 8/8 [==============================] - 0s 11ms/step - loss: 16.3554 - val_loss: 17.1801 Epoch 10/100 8/8 [==============================] - 0s 10ms/step - loss: 15.5110 - val_loss: 16.4358 Epoch 11/100 8/8 [==============================] - 0s 10ms/step - loss: 14.7199 - val_loss: 15.6965 Epoch 12/100 8/8 [==============================] - 0s 11ms/step - loss: 13.8763 - val_loss: 14.9948 Epoch 13/100 8/8 [==============================] - 0s 11ms/step - loss: 13.0945 - val_loss: 14.2853 Epoch 14/100 8/8 [==============================] - 0s 10ms/step - loss: 12.2870 - val_loss: 13.5716 Epoch 15/100 8/8 [==============================] - 0s 13ms/step - loss: 11.4837 - val_loss: 12.7729 Epoch 16/100 8/8 [==============================] - 0s 10ms/step - loss: 10.7063 - val_loss: 12.0427 Epoch 17/100 8/8 [==============================] - 0s 12ms/step - loss: 9.8979 - val_loss: 11.2940 Epoch 18/100 8/8 [==============================] - 0s 9ms/step - loss: 9.1303 - val_loss: 10.5994 Epoch 19/100 8/8 [==============================] - 0s 22ms/step - loss: 8.3971 - val_loss: 9.8753 Epoch 20/100 8/8 [==============================] - 0s 21ms/step - loss: 7.6491 - val_loss: 9.1706 Epoch 21/100 8/8 [==============================] - 0s 12ms/step - loss: 6.9654 - val_loss: 8.4357 Epoch 22/100 8/8 [==============================] - 0s 9ms/step - loss: 6.3487 - val_loss: 7.8406 Epoch 23/100 8/8 [==============================] - 0s 10ms/step - loss: 5.7869 - val_loss: 7.1229 Epoch 24/100 8/8 [==============================] - 0s 14ms/step - loss: 5.3087 - val_loss: 6.4682 Epoch 25/100 8/8 [==============================] - 0s 10ms/step - loss: 4.8190 - val_loss: 5.9280 Epoch 26/100 8/8 [==============================] - 0s 9ms/step - loss: 4.3753 - val_loss: 5.3652 Epoch 27/100 8/8 [==============================] - 0s 12ms/step - loss: 3.9553 - val_loss: 4.6350 Epoch 28/100 8/8 [==============================] - 0s 16ms/step - loss: 3.5127 - val_loss: 4.0965 Epoch 29/100 8/8 [==============================] - 0s 12ms/step - loss: 3.1933 - val_loss: 3.4377 Epoch 30/100 8/8 [==============================] - 0s 12ms/step - loss: 2.8976 - val_loss: 3.2094 Epoch 31/100 8/8 [==============================] - 0s 14ms/step - loss: 2.7605 - val_loss: 2.8405 Epoch 32/100 8/8 [==============================] - 0s 11ms/step - loss: 2.6630 - val_loss: 2.7411 Epoch 33/100 8/8 [==============================] - 0s 19ms/step - loss: 2.5866 - val_loss: 2.6966 Epoch 34/100 8/8 [==============================] - 0s 12ms/step - loss: 2.5401 - val_loss: 2.6735 Epoch 35/100 8/8 [==============================] - 0s 10ms/step - loss: 2.5222 - val_loss: 2.6101 Epoch 36/100 8/8 [==============================] - 0s 22ms/step - loss: 2.4940 - val_loss: 2.5362 Epoch 37/100 8/8 [==============================] - 0s 14ms/step - loss: 2.5069 - val_loss: 2.5364 Epoch 38/100 8/8 [==============================] - 0s 11ms/step - loss: 2.4789 - val_loss: 2.4968 Epoch 39/100 8/8 [==============================] - 0s 10ms/step - loss: 2.5001 - val_loss: 2.4952 Epoch 40/100 8/8 [==============================] - 0s 14ms/step - loss: 2.4909 - val_loss: 2.4861 Epoch 41/100 8/8 [==============================] - 0s 15ms/step - loss: 2.4876 - val_loss: 2.4934 Epoch 42/100 8/8 [==============================] - 0s 11ms/step - loss: 2.5050 - val_loss: 2.4709 Epoch 43/100 8/8 [==============================] - 0s 14ms/step - loss: 2.4897 - val_loss: 2.4948 Epoch 44/100 8/8 [==============================] - 0s 12ms/step - loss: 2.4835 - val_loss: 2.4711 Epoch 45/100 8/8 [==============================] - 0s 7ms/step - loss: 2.5012 - val_loss: 2.4704 Epoch 46/100 8/8 [==============================] - 0s 19ms/step - loss: 2.4887 - val_loss: 2.4912 Epoch 47/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4848 - val_loss: 2.4834 Epoch 48/100 8/8 [==============================] - 0s 19ms/step - loss: 2.4970 - val_loss: 2.4806 Epoch 49/100 8/8 [==============================] - 0s 11ms/step - loss: 2.5046 - val_loss: 2.4756 Epoch 50/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4840 - val_loss: 2.5091 Epoch 51/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4843 - val_loss: 2.4974 Epoch 52/100 8/8 [==============================] - 0s 14ms/step - loss: 2.4747 - val_loss: 2.5129 Epoch 53/100 8/8 [==============================] - 0s 13ms/step - loss: 2.4773 - val_loss: 2.5032 Epoch 54/100 8/8 [==============================] - 0s 13ms/step - loss: 2.4973 - val_loss: 2.4971 Epoch 55/100 8/8 [==============================] - 0s 12ms/step - loss: 2.4763 - val_loss: 2.5366 Epoch 56/100 8/8 [==============================] - 0s 12ms/step - loss: 2.4699 - val_loss: 2.4978 Epoch 57/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4667 - val_loss: 2.4884 Epoch 58/100 8/8 [==============================] - 0s 12ms/step - loss: 2.4843 - val_loss: 2.4784 Epoch 59/100 8/8 [==============================] - 0s 13ms/step - loss: 2.4812 - val_loss: 2.4895 Epoch 60/100 8/8 [==============================] - 0s 7ms/step - loss: 2.4700 - val_loss: 2.4906 Epoch 61/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4737 - val_loss: 2.4932 Epoch 62/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4783 - val_loss: 2.5025 Epoch 63/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4666 - val_loss: 2.4894 Epoch 64/100 8/8 [==============================] - 0s 7ms/step - loss: 2.4691 - val_loss: 2.4728 Epoch 65/100 8/8 [==============================] - 0s 11ms/step - loss: 2.4670 - val_loss: 2.4843 Epoch 66/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4942 - val_loss: 2.4673 Epoch 67/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4759 - val_loss: 2.4894 Epoch 68/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4764 - val_loss: 2.4699 Epoch 69/100 8/8 [==============================] - 0s 11ms/step - loss: 2.4781 - val_loss: 2.4794 Epoch 70/100 8/8 [==============================] - 0s 11ms/step - loss: 2.4698 - val_loss: 2.5104 Epoch 71/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4804 - val_loss: 2.4731 Epoch 72/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4730 - val_loss: 2.4873 Epoch 73/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4809 - val_loss: 2.4709 Epoch 74/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4758 - val_loss: 2.4974 Epoch 75/100 8/8 [==============================] - 0s 13ms/step - loss: 2.4828 - val_loss: 2.4708 Epoch 76/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4730 - val_loss: 2.4615 Epoch 77/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4888 - val_loss: 2.4951 Epoch 78/100 8/8 [==============================] - 0s 11ms/step - loss: 2.4851 - val_loss: 2.4576 Epoch 79/100 8/8 [==============================] - 0s 14ms/step - loss: 2.4842 - val_loss: 2.4686 Epoch 80/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4889 - val_loss: 2.4618 Epoch 81/100 8/8 [==============================] - 0s 13ms/step - loss: 2.4898 - val_loss: 2.4714 Epoch 82/100 8/8 [==============================] - 0s 16ms/step - loss: 2.4957 - val_loss: 2.4495 Epoch 83/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4691 - val_loss: 2.5070 Epoch 84/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4927 - val_loss: 2.4614 Epoch 85/100 8/8 [==============================] - 0s 8ms/step - loss: 2.5247 - val_loss: 2.4874 Epoch 86/100 8/8 [==============================] - 0s 8ms/step - loss: 2.5028 - val_loss: 2.5083 Epoch 87/100 8/8 [==============================] - 0s 9ms/step - loss: 2.5116 - val_loss: 2.4441 Epoch 88/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4629 - val_loss: 2.5328 Epoch 89/100 8/8 [==============================] - 0s 11ms/step - loss: 2.5012 - val_loss: 2.4583 Epoch 90/100 8/8 [==============================] - 0s 10ms/step - loss: 2.4791 - val_loss: 2.4591 Epoch 91/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4631 - val_loss: 2.4995 Epoch 92/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4706 - val_loss: 2.4477 Epoch 93/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4956 - val_loss: 2.4648 Epoch 94/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4805 - val_loss: 2.4921 Epoch 95/100 8/8 [==============================] - 0s 7ms/step - loss: 2.4781 - val_loss: 2.4588 Epoch 96/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4714 - val_loss: 2.4745 Epoch 97/100 8/8 [==============================] - 0s 9ms/step - loss: 2.4776 - val_loss: 2.4915 Epoch 98/100 8/8 [==============================] - 0s 8ms/step - loss: 2.4731 - val_loss: 2.4574 Epoch 99/100 8/8 [==============================] - 0s 12ms/step - loss: 2.4996 - val_loss: 2.4559 Epoch 100/100 8/8 [==============================] - 0s 8ms/step - loss: 2.5075 - val_loss: 2.4584
2023/01/31 19:15:05 WARNING mlflow.tensorflow: Failed to infer model signature: could not sample data to infer model signature: tuple index out of range 2023/01/31 19:15:05 WARNING mlflow.tensorflow: You are saving a TensorFlow Core model or Keras model without a signature. Inference with mlflow.pyfunc.spark_udf() will not work unless the model's pyfunc representation accepts pandas DataFrames as inference inputs. WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmp06tnowpp/model/data/model/assets
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmp06tnowpp/model/data/model/assets 2023/01/31 19:15:05 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmp06tnowpp/model, flavor: tensorflow), fall back to return ['tensorflow==2.11.0']. Set logging level to DEBUG to see the full traceback.
CPU times: user 8.24 s, sys: 846 ms, total: 9.09 s Wall time: 10.7 s
history object:hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
hist.tail()
| loss | val_loss | epoch | |
|---|---|---|---|
| 95 | 2.471398 | 2.474460 | 95 |
| 96 | 2.477598 | 2.491461 | 96 |
| 97 | 2.473082 | 2.457406 | 97 |
| 98 | 2.499563 | 2.455935 | 98 |
| 99 | 2.507488 | 2.458402 | 99 |
def plot_loss(history):
plt.plot(history.history['loss'], label='loss')
plt.plot(history.history['val_loss'], label='val_loss')
plt.ylim([0, 10])
plt.xlabel('Epoch')
plt.ylabel('Error [MPG]')
plt.legend()
plt.grid(True)
Use plot_loss(history) provided to visualize the progression in loss function for training and validation data sets.
plot_loss(history)# YOUR CODE HERE
Model.evaluate()test_results = {}
test_results['linear_model'] = linear_model.evaluate(x=test_features, y=test_labels)# YOUR CODE HERE
3/3 [==============================] - 0s 6ms/step - loss: 2.4612
test_results
{'linear_model': 2.4612009525299072}
You just implemented a linear model for multiple inputs. Now, you are ready to implement multiple-input DNN models.
The code is very similar except the model is expanded to include some "hidden" non-linear layers. The name "hidden" here just means not directly connected to the inputs or outputs.
normalizer for a multiple-input model).Dense layers with the ReLU (relu) activation function nonlinearity. One way is to set parameter activation inside Dense Set the number of neurons at each layer to be 64. Dense single-output layer. compile method in the build_and_compile_model function below. def build_and_compile_model(norm):
model = tf.keras.Sequential([
norm,
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1),
]) # YOUR CODE HERE
model.compile(loss='mean_absolute_error',
optimizer=tf.keras.optimizers.Adam())
return model
normalizer (defined earlier) as the normalization layer:dnn_model = build_and_compile_model(normalizer)# YOUR CODE HERE
Model.summary(). This model has quite a few more trainable parameters than the linear models:dnn_model.summary()# YOUR CODE HERE
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
normalization (Normalizatio (None, 9) 19
n)
dense_2 (Dense) (None, 64) 640
dense_3 (Dense) (None, 64) 4160
dense_4 (Dense) (None, 1) 65
=================================================================
Total params: 4,884
Trainable params: 4,865
Non-trainable params: 19
_________________________________________________________________
Model.fit:%%time
history = dnn_model.fit(
train_features,
train_labels,
validation_split=0.2,
verbose=0, epochs=100)
2023/01/31 19:28:25 INFO mlflow.utils.autologging_utils: Created MLflow autologging run with ID '74c24586ef67442d973256f1c6733911', which will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow 2023/01/31 19:28:33 WARNING mlflow.tensorflow: Failed to infer model signature: could not sample data to infer model signature: Cannot log input example or model signature for input with type <class 'pandas.core.frame.DataFrame'>. TensorFlow Keras autologging can only log input examples and model signatures for the following input types: numpy.ndarray, dict[string -> numpy.ndarray], tensorflow.keras.utils.Sequence, and tensorflow.data.Dataset (TensorFlow >= 2.1.0 required) 2023/01/31 19:28:33 WARNING mlflow.tensorflow: You are saving a TensorFlow Core model or Keras model without a signature. Inference with mlflow.pyfunc.spark_udf() will not work unless the model's pyfunc representation accepts pandas DataFrames as inference inputs. WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpubdm01t3/model/data/model/assets
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpubdm01t3/model/data/model/assets 2023/01/31 19:28:34 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpubdm01t3/model, flavor: tensorflow), fall back to return ['tensorflow==2.11.0']. Set logging level to DEBUG to see the full traceback.
CPU times: user 7.79 s, sys: 659 ms, total: 8.45 s Wall time: 9.11 s
plot_loss(history)
Do you think the DNN model is overfitting? What gives away?
As the validation does not keep improving and the training keeps improving it is an ok model If the training and validation were both improving that would mean there is no difference and then it is overfitting
test_results['dnn_model'] = dnn_model.evaluate(test_features, test_labels, verbose=0)
test_results
{'linear_model': 2.4612009525299072, 'dnn_model': 1.6599880456924438}
pd.DataFrame(test_results, index=['Mean absolute error [MPG]']).T
| Mean absolute error [MPG] | |
|---|---|
| linear_model | 2.461201 |
| dnn_model | 1.659988 |
These results match the validation error observed during training.
dnn_model on the test set using Keras Model.predict and review the loss. Use .flatten().test_predictions = dnn_model.predict(test_features) # YOUR CODE HERE
a = plt.axes(aspect='equal')
plt.scatter(test_labels, test_predictions)
plt.xlabel('True Values [MPG]')
plt.ylabel('Predictions [MPG]')
lims = [0, 50]
plt.xlim(lims)
plt.ylim(lims)
_ = plt.plot(lims, lims)
3/3 [==============================] - 0s 2ms/step
error = (test_labels - test_predictions.squeeze()) # YOUR CODE HERE
plt.hist(error, bins=25)
plt.xlabel('Prediction Error [MPG]')
_ = plt.ylabel('Count')
Model.save:dnn_model.save('dnn_model')
WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: dnn_model/assets
INFO:tensorflow:Assets written to: dnn_model/assets
Model.load_model; it gives identical output:from tensorflow import keras
reloaded = keras.models.load_model('dnn_model') # YOUR CODE HERE
test_results['reloaded'] = reloaded.evaluate(
test_features, test_labels, verbose=0)
pd.DataFrame(test_results, index=['Mean absolute error [MPG]']).T
| Mean absolute error [MPG] | |
|---|---|
| linear_model | 2.461201 |
| dnn_model | 1.659988 |
| reloaded | 1.659988 |
We mentioned that the relu activation function introduce non-linearity; let's visualize it. Since there are six numerical features and 1 categorical features, it is impossible to plot all the dimensions on a 2D plot; we need to simplify/isolate it.
Note: in this task, code is provided; the focus in on understanding.
We focus on the relationship between feature Displacement and target MPG.
To do so, create a new dataset of the same size as train_features, but all other features are set at their median values; then set the Displacement between 0 and 500.
fake = np.outer(np.ones(train_features.shape[0]), train_features.median())
fake = pd.DataFrame(fake, columns = train_features.columns)
fake.Displacement = np.linspace(0, 500, train_features.shape[0])
Create a plotting function to:
a) visualize real values between Displacement and MPG from the training dataset in scatter plot
b) overlay the predicted MPG from Displacement varying from 0 to 500, but holding all other features constant.
def plot_displacement(x, y):
plt.scatter(train_features['Displacement'], train_labels, label='Data')
plt.plot(x, y, color='k', label='Predictions')
plt.xlabel('Displacement')
plt.ylabel('MPG')
plt.legend()
plot_displacement(fake.Displacement, linear_model(fake))
plot_displacement(fake.Displacement, dnn_model.predict(fake))
10/10 [==============================] - 0s 3ms/step
What are the other activation functions? Check the list of activations.
Optional. Modify the DNN model with a different activation function, and fit it on the data; does it perform better?
def build_and_compile_model2(norm):
model = tf.keras.Sequential([
norm,
tf.keras.layers.Dense(64, activation='tanh'),
tf.keras.layers.Dense(64, activation='tanh'),
tf.keras.layers.Dense(1),
])
model.compile(loss='mean_absolute_error',
optimizer=tf.keras.optimizers.Adam())
return model
dnn_model2 = build_and_compile_model2(normalizer)
dnn_model2.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
normalization (Normalizatio (None, 9) 19
n)
dense_5 (Dense) (None, 64) 640
dense_6 (Dense) (None, 64) 4160
dense_7 (Dense) (None, 1) 65
=================================================================
Total params: 4,884
Trainable params: 4,865
Non-trainable params: 19
_________________________________________________________________
%%time
history2 = dnn_model2.fit(
train_features,
train_labels,
validation_split=0.2,
verbose=0, epochs=100)
2023/01/31 20:05:42 INFO mlflow.utils.autologging_utils: Created MLflow autologging run with ID '028b8d02ca534781996b9d0258219ec8', which will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow 2023/01/31 20:05:49 WARNING mlflow.tensorflow: Failed to infer model signature: could not sample data to infer model signature: Cannot log input example or model signature for input with type <class 'pandas.core.frame.DataFrame'>. TensorFlow Keras autologging can only log input examples and model signatures for the following input types: numpy.ndarray, dict[string -> numpy.ndarray], tensorflow.keras.utils.Sequence, and tensorflow.data.Dataset (TensorFlow >= 2.1.0 required) 2023/01/31 20:05:49 WARNING mlflow.tensorflow: You are saving a TensorFlow Core model or Keras model without a signature. Inference with mlflow.pyfunc.spark_udf() will not work unless the model's pyfunc representation accepts pandas DataFrames as inference inputs. WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpzz_6bi0u/model/data/model/assets
INFO:tensorflow:Assets written to: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpzz_6bi0u/model/data/model/assets 2023/01/31 20:05:50 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /var/folders/dx/8bzgllmx7g13k02q2z8660kr0000gq/T/tmpzz_6bi0u/model, flavor: tensorflow), fall back to return ['tensorflow==2.11.0']. Set logging level to DEBUG to see the full traceback.
CPU times: user 7.6 s, sys: 633 ms, total: 8.23 s Wall time: 8.1 s
plot_loss(history2)
test_results['dnn_model2'] = dnn_model2.evaluate(test_features, test_labels, verbose=0)
pd.DataFrame(test_results, index=['Mean absolute error [MPG]']).T
| Mean absolute error [MPG] | |
|---|---|
| linear_model | 2.461201 |
| dnn_model | 1.659988 |
| reloaded | 1.659988 |
| dnn_model2 | 2.289334 |
dnn_model2.save('dnn_model2')
WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: dnn_model2/assets
INFO:tensorflow:Assets written to: dnn_model2/assets
In this task, we briefly explore MLflow Tracking, one of four primary functions that MLflow offers for managing the end-to-end machine learning lifecycle. We will access the information runs programmatically in python and then set up the MLflow UI for easy interaction.
Experiments.
MLflow Tracking is organized around the concept of runs, which are executions of some piece of modeling code; and runs are organized into experiments.
We set the auto logging in the beginning, we can verify that
0file:///content/mlruns/0 in Google Drive.from mlflow import MlflowClient
client = MlflowClient()
client.search_experiments()
#client.list_experiments() old code
[<Experiment: artifact_location='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0', creation_time=1675008942937, experiment_id='0', last_update_time=1675008942937, lifecycle_stage='active', name='Default', tags={}>]
Runs.
List information for runs that are under experiment '0' using mlflow.list_run_infos().
client.search_runs('0')# YOUR CODE HERE
[<Run: data=<RunData: metrics={'loss': 2.2992823123931885, 'val_loss': 2.5714523792266846}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.001',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "028b8d02ca534781996b9d0258219ec8", '
'"artifact_path": "model", "utc_time_created": '
'"2023-02-01 02:05:49.998465", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"65cdff00373b4843a75d01996e8f9e8c", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'big-calf-905',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/028b8d02ca534781996b9d0258219ec8/artifacts', end_time=1675217150872, experiment_id='0', lifecycle_stage='active', run_id='028b8d02ca534781996b9d0258219ec8', run_name='big-calf-905', run_uuid='028b8d02ca534781996b9d0258219ec8', start_time=1675217142792, status='FINISHED', user_id='Gilles'>>,
<Run: data=<RunData: metrics={'loss': 1.6439439058303833, 'val_loss': 2.128159761428833}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.001',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "74c24586ef67442d973256f1c6733911", '
'"artifact_path": "model", "utc_time_created": '
'"2023-02-01 01:28:33.418657", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"1b96948555444e7a9046f8ef82d32ccd", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'rebellious-skink-608',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/74c24586ef67442d973256f1c6733911/artifacts', end_time=1675214914366, experiment_id='0', lifecycle_stage='active', run_id='74c24586ef67442d973256f1c6733911', run_name='rebellious-skink-608', run_uuid='74c24586ef67442d973256f1c6733911', start_time=1675214905270, status='FINISHED', user_id='Gilles'>>,
<Run: data=<RunData: metrics={'loss': 2.5074875354766846, 'val_loss': 2.458402395248413}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.1',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "416c1ca789a646cba1604f5d1ec01953", '
'"artifact_path": "model", "utc_time_created": '
'"2023-02-01 01:15:05.201458", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"2667c26764b9499fb31439840731eaf4", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'glamorous-wolf-723',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/416c1ca789a646cba1604f5d1ec01953/artifacts', end_time=1675214105756, experiment_id='0', lifecycle_stage='active', run_id='416c1ca789a646cba1604f5d1ec01953', run_name='glamorous-wolf-723', run_uuid='416c1ca789a646cba1604f5d1ec01953', start_time=1675214095037, status='FINISHED', user_id='Gilles'>>,
<Run: data=<RunData: metrics={'loss': 2.465920925140381, 'val_loss': 2.5140719413757324}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.1',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "b2d64a38d9cb4809ac02748aae99134f", '
'"artifact_path": "model", "utc_time_created": '
'"2023-02-01 00:33:48.326965", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"d9e03f5a3730471c9c070d3d87027490", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'rogue-bear-504',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/b2d64a38d9cb4809ac02748aae99134f/artifacts', end_time=1675211629466, experiment_id='0', lifecycle_stage='active', run_id='b2d64a38d9cb4809ac02748aae99134f', run_name='rogue-bear-504', run_uuid='b2d64a38d9cb4809ac02748aae99134f', start_time=1675211617714, status='FINISHED', user_id='Gilles'>>,
<Run: data=<RunData: metrics={'loss': 2.472140312194824, 'val_loss': 2.522207021713257}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.1',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "e8b1a5a4554d41d4b7fb013f70c098a4", '
'"artifact_path": "model", "utc_time_created": '
'"2023-01-29 16:15:54.389769", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"8b90b23d6693471db9012ce2947bddfe", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'mercurial-hare-193',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/e8b1a5a4554d41d4b7fb013f70c098a4/artifacts', end_time=1675009009877, experiment_id='0', lifecycle_stage='active', run_id='e8b1a5a4554d41d4b7fb013f70c098a4', run_name='mercurial-hare-193', run_uuid='e8b1a5a4554d41d4b7fb013f70c098a4', start_time=1675008943198, status='FINISHED', user_id='Gilles'>>]
mlflow.last_active_run() autolog_run = mlflow.last_active_run()# YOUR CODE HERE
print_auto_logged_info provided below to fetch the auto logged parameters and metrics for autolog_run. import json
def print_auto_logged_info(r):
tags = {k: v for k, v in r.data.tags.items() if not k.startswith("mlflow.")}
artifacts = [f.path for f in MlflowClient().list_artifacts(r.info.run_id, "model")]
print("run_id: {}".format(r.info.run_id))
print("artifacts: {}".format(artifacts))
print("params: {}".format(json.dumps(r.data.params, indent=4)))
print("metrics: {}".format(r.data.metrics))
print("tags: {}".format(tags))
print_auto_logged_info(
autolog_run# YOUR CODE HERE
)
run_id: 028b8d02ca534781996b9d0258219ec8
artifacts: ['model/.DS_Store', 'model/MLmodel', 'model/conda.yaml', 'model/data', 'model/python_env.yaml', 'model/requirements.txt']
params: {
"opt_ema_momentum": "0.99",
"opt_epsilon": "1e-07",
"opt_clipvalue": "None",
"validation_freq": "1",
"validation_steps": "None",
"shuffle": "True",
"use_multiprocessing": "False",
"opt_jit_compile": "False",
"opt_is_legacy_optimizer": "False",
"sample_weight": "None",
"initial_epoch": "0",
"max_queue_size": "10",
"validation_batch_size": "None",
"class_weight": "None",
"opt_beta_2": "0.999",
"validation_split": "0.2",
"opt_weight_decay": "None",
"steps_per_epoch": "None",
"epochs": "100",
"opt_name": "Adam",
"opt_amsgrad": "False",
"opt_use_ema": "False",
"opt_clipnorm": "None",
"batch_size": "None",
"workers": "1",
"opt_learning_rate": "0.001",
"opt_global_clipnorm": "None",
"opt_ema_overwrite_frequency": "None",
"opt_beta_1": "0.9"
}
metrics: {'val_loss': 2.5714523792266846, 'loss': 2.2992823123931885}
tags: {}
runs = MlflowClient().search_runs(experiment_ids = ['0'], order_by=['metrics.val_loss']) # YOUR CODE HERE
runs[0]
<Run: data=<RunData: metrics={'loss': 1.6439439058303833, 'val_loss': 2.128159761428833}, params={'batch_size': 'None',
'class_weight': 'None',
'epochs': '100',
'initial_epoch': '0',
'max_queue_size': '10',
'opt_amsgrad': 'False',
'opt_beta_1': '0.9',
'opt_beta_2': '0.999',
'opt_clipnorm': 'None',
'opt_clipvalue': 'None',
'opt_ema_momentum': '0.99',
'opt_ema_overwrite_frequency': 'None',
'opt_epsilon': '1e-07',
'opt_global_clipnorm': 'None',
'opt_is_legacy_optimizer': 'False',
'opt_jit_compile': 'False',
'opt_learning_rate': '0.001',
'opt_name': 'Adam',
'opt_use_ema': 'False',
'opt_weight_decay': 'None',
'sample_weight': 'None',
'shuffle': 'True',
'steps_per_epoch': 'None',
'use_multiprocessing': 'False',
'validation_batch_size': 'None',
'validation_freq': '1',
'validation_split': '0.2',
'validation_steps': 'None',
'workers': '1'}, tags={'mlflow.autologging': 'tensorflow',
'mlflow.log-model.history': '[{"run_id": "74c24586ef67442d973256f1c6733911", '
'"artifact_path": "model", "utc_time_created": '
'"2023-02-01 01:28:33.418657", "flavors": '
'{"tensorflow": {"code": null, "data": "data", '
'"model_type": "keras", "keras_version": '
'"2.11.0", "save_format": "tf"}, '
'"python_function": {"loader_module": '
'"mlflow.tensorflow", "python_version": "3.8.15", '
'"data": "data", "env": {"conda": "conda.yaml", '
'"virtualenv": "python_env.yaml"}}}, '
'"model_uuid": '
'"1b96948555444e7a9046f8ef82d32ccd", '
'"mlflow_version": "2.1.1"}]',
'mlflow.runName': 'rebellious-skink-608',
'mlflow.source.name': '/Users/Gilles/anaconda3/envs/sa/lib/python3.8/site-packages/ipykernel_launcher.py',
'mlflow.source.type': 'LOCAL',
'mlflow.user': 'Gilles'}>, info=<RunInfo: artifact_uri='file:///Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/mlruns/0/74c24586ef67442d973256f1c6733911/artifacts', end_time=1675214914366, experiment_id='0', lifecycle_stage='active', run_id='74c24586ef67442d973256f1c6733911', run_name='rebellious-skink-608', run_uuid='74c24586ef67442d973256f1c6733911', start_time=1675214905270, status='FINISHED', user_id='Gilles'>>
To see what's logged in the file system /content/mlruns/, click tab files in the left sidepanel in Colab. For example,
mlruns
└── 0
├── 3a5aebdd35ef46fb8dc35b40e542f0a4
│ ├── artifacts
│ ├── meta.yaml
│ ├── metrics
│ ├── params
│ └── tags
├── c627bc526c4a4c418a8285627e61a16d
│ ├── artifacts
│ ├── meta.yaml
│ ├── metrics
│ ├── params
│ └── tags
└── meta.yaml
11 directories, 3 files
Inspect the model summary of the DNN model you ran previously; it is located at artifacts/model_summary.txt of the corresponding run. Use cat $filepath.
!ls mlruns/0/{runs[0].info.run_id}/artifacts/model_summary.txt
mlruns/0/74c24586ef67442d973256f1c6733911/artifacts/model_summary.txt
!cat mlruns/0/74c24586ef67442d973256f1c6733911/artifacts/model_summary.txt# YOUR CODE HERE (bash)
zsh:1: unknown file attribute: b
from IPython.display import Image, display
# change the filename to wherever you downloaded/uploaded the file
filename = '../img/summary.jpg'
display(Image(filename=filename))
filename = '../img/mlrunpic1.png'
display(Image(filename=filename))
from IPython.display import Image, display
# change the filename to wherever you downloaded/uploaded the file
filename = '../img/mlrunpic2.jpg'
display(Image(filename=filename))
from IPython.display import Image, display
# change the filename to wherever you downloaded/uploaded the file
filename = '../img/mlrunpic3.jpg'
display(Image(filename=filename))
# the 4 next experiments (linear, redo dnn, dnn2)
from IPython.display import Image, display
# change the filename to wherever you downloaded/uploaded the file
filename = '../img/dnnModel.jpg'
display(Image(filename=filename))
Tracking UI.
MLflow provides an UI for us to visualize, search and compare runs, as well as download run artifacts or metadata for analysis in other tools.
If your runs are logged to a local mlruns directory, run mlflow ui in the directory above it will load the corresponding runs.
Running localhost server in Colab, however, requires a bit of extra work:
NOTE. NEVER share your secrets. Best to keep NGROK_AUTH_TOKEN as an environment variable and retrieve it via os.environ.get("NGROK_AUTH_TOKEN").
# run tracking UI in the background
get_ipython().system_raw("mlflow ui --port 5000 &")
# create remote tunnel using ngrok.com to allow local port access
from pyngrok import ngrok
# Terminate open tunnels if exist
ngrok.kill()
# Setting the authtoken (see Note above)
NGROK_AUTH_TOKEN = "2L0umvOgkhKcI2PiJKkSWQuvvSI_42VfXjNJSxjVrUzCrKyhU" # YOUR CODE HERE
ngrok.set_auth_token(NGROK_AUTH_TOKEN)
# Open an HTTPs tunnel on port 5000 for http://localhost:5000
ngrok_tunnel = ngrok.connect(addr="5000", proto="http", bind_tls=True)
print("MLflow Tracking UI:", ngrok_tunnel.public_url)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) Input In [99], in <cell line: 5>() 2 get_ipython().system_raw("mlflow ui --port 5000 &") 4 # create remote tunnel using ngrok.com to allow local port access ----> 5 from pyngrok import ngrok 6 # Terminate open tunnels if exist 7 ngrok.kill() ModuleNotFoundError: No module named 'pyngrok'
[2023-01-31 21:02:50 -0600] [66666] [INFO] Starting gunicorn 20.1.0 [2023-01-31 21:02:50 -0600] [66666] [INFO] Listening at: http://127.0.0.1:5000 (66666) [2023-01-31 21:02:50 -0600] [66666] [INFO] Using worker: sync [2023-01-31 21:02:50 -0600] [66669] [INFO] Booting worker with pid: 66669 [2023-01-31 21:02:50 -0600] [66670] [INFO] Booting worker with pid: 66670 [2023-01-31 21:02:50 -0600] [66671] [INFO] Booting worker with pid: 66671 [2023-01-31 21:02:50 -0600] [66672] [INFO] Booting worker with pid: 66672
Interact with Tracking UI.
Open the link, output from the previous cell. get oriented, Parameters, Metrics, Artifacts, and so on.
When you are done, make sure to terminate the open tunnel:
filename = '../img/mlflowexp1.jpg'
display(Image(filename=filename))
filename = '../img/mlflowexp2.jpg'
display(Image(filename=filename))
filename = '../img/mlflowexp3.jpg'
display(Image(filename=filename))
filename = '../img/mlfowexp4.jpg'
display(Image(filename=filename))
#http://127.0.0.1:5000/#/experiments/0/runs/028b8d02ca534781996b9d0258219ec8
# stop the kernel
ngrok.kill()
--------------------------------------------------------------------------- NameError Traceback (most recent call last) Input In [100], in <cell line: 1>() ----> 1 ngrok.kill() NameError: name 'ngrok' is not defined
Instantiate and train a TPOT auto-ML regressor.
The parameters are set fairly arbitrarily (if time permits, you shall experiment with different sets of parameters after reading what each parameter does). Use these parameter values:
generations: 10
population_size: 40
scoring: negative mean absolute error; read more in scoring functions in TPOT
verbosity: 2 (so you can see each generation's performance)
The final line with create a Python script tpot_products_pipeline.py with the code to create the optimal model found by TPOT.
%%time
from tpot import TPOTRegressor
tpot = TPOTRegressor(generations=10,
population_size=40,
scoring=None, # YOUR CODE HERE scoring = 'f1' does not seem to do anything better
verbosity=2,
random_state=42)
tpot.fit(train_features, train_labels)
print(f"Tpop score on test data: {tpot.score(test_features, test_labels):.2f}")
tpot.export('tpot_mpg_pipeline.py')
Generation 1 - Current best internal CV score: -8.077745295810276 Generation 2 - Current best internal CV score: -8.077745295810276 Generation 3 - Current best internal CV score: -8.077745295810276 Generation 4 - Current best internal CV score: -8.00800302618326 Generation 5 - Current best internal CV score: -8.00800302618326 Generation 6 - Current best internal CV score: -7.827227772486582 Generation 7 - Current best internal CV score: -7.82722777248655 Generation 8 - Current best internal CV score: -7.82722777248655 Generation 9 - Current best internal CV score: -7.82722777248655 Generation 10 - Current best internal CV score: -7.827227772486549 Best pipeline: LassoLarsCV(ExtraTreesRegressor(MaxAbsScaler(VarianceThreshold(input_matrix, threshold=0.0005)), bootstrap=True, max_features=0.9500000000000001, min_samples_leaf=14, min_samples_split=6, n_estimators=100), normalize=True) Tpop score on test data: -7.40 CPU times: user 10min 19s, sys: 7.97 s, total: 10min 27s Wall time: 8min 11s
sklearn.metrics.SCORERS is deprecated and will be removed in v1.3. Please use sklearn.metrics.get_scorer_names to get a list of available scorers and sklearn.metrics.get_metric to get scorer.
import sklearn.metrics
sklearn.metrics.get_scorer_names()
['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'balanced_accuracy', 'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score', 'jaccard', 'jaccard_macro', 'jaccard_micro', 'jaccard_samples', 'jaccard_weighted', 'matthews_corrcoef', 'max_error', 'mutual_info_score', 'neg_brier_score', 'neg_log_loss', 'neg_mean_absolute_error', 'neg_mean_absolute_percentage_error', 'neg_mean_gamma_deviance', 'neg_mean_poisson_deviance', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'neg_median_absolute_error', 'neg_negative_likelihood_ratio', 'neg_root_mean_squared_error', 'normalized_mutual_info_score', 'positive_likelihood_ratio', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'rand_score', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc', 'roc_auc_ovo', 'roc_auc_ovo_weighted', 'roc_auc_ovr', 'roc_auc_ovr_weighted', 'top_k_accuracy', 'v_measure_score']
sklearn.metrics.get_scorer('f1')
make_scorer(f1_score, average=binary)
dataset.head(2)
| MPG | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model Year | Europe | Japan | USA | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 18.0 | 8 | 307.0 | 130.0 | 3504.0 | 12.0 | 70 | 0 | 0 | 1 |
| 1 | 15.0 | 8 | 350.0 | 165.0 | 3693.0 | 11.5 | 70 | 0 | 0 | 1 |
Examine the model pipeline that TPOT regressor offers. If you see any model, function, or class that are not familiar, look them up!
Note: There is randomness to the way the TPOT searches, so it's possible you won't have exactly the same result as your classmate.
cat tpot_mpg_pipeline.py
import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.feature_selection import VarianceThreshold
from sklearn.linear_model import LassoLarsCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from sklearn.preprocessing import MaxAbsScaler
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive
# NOTE: Make sure that the outcome column is labeled 'target' in the data file
tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1)
training_features, testing_features, training_target, testing_target = \
train_test_split(features, tpot_data['target'], random_state=42)
# Average CV score on the training set was: -7.827227772486549
exported_pipeline = make_pipeline(
VarianceThreshold(threshold=0.0005),
MaxAbsScaler(),
StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=True, max_features=0.9500000000000001, min_samples_leaf=14, min_samples_split=6, n_estimators=100)),
LassoLarsCV(normalize=True)
)
# Fix random state for all the steps in exported pipeline
set_param_recursive(exported_pipeline.steps, 'random_state', 42)
exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)
tpot_mpg_pipeline.py to build a model on our training set and make predictions on the test set.
Save the predictions as y_pred, and compute appropriate evaluation metric.
You may find that for this simple data set, the neural network we built outperforms the tree-based model, yet note it is not a conclusion that we can generalize for all tabular data. import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.feature_selection import VarianceThreshold
from sklearn.linear_model import LassoLarsCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from sklearn.preprocessing import MaxAbsScaler
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive
####tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
tpot_data = dataset
features = tpot_data.drop('MPG', axis=1)
training_features, testing_features, training_target, testing_target = \
train_test_split(features, tpot_data['MPG'], random_state=42)
exported_pipeline = make_pipeline(
VarianceThreshold(threshold=0.0005),
MaxAbsScaler(),
StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=True, max_features=0.9500000000000001, min_samples_leaf=14, min_samples_split=6, n_estimators=100)),
LassoLarsCV(normalize=True)
)
# Fix random state for all the steps in exported pipeline
set_param_recursive(exported_pipeline.steps, 'random_state', 42)
exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)
'normalize' was deprecated in version 1.2 and will be removed in 1.4.
If you wish to scale the data, use Pipeline with a StandardScaler in a preprocessing stage. To reproduce the previous behavior:
from sklearn.pipeline import make_pipeline
model = make_pipeline(StandardScaler(with_mean=False), LassoLarsCV())
If you wish to pass a sample_weight parameter, you need to pass it as a fit parameter to each step of the pipeline as follows:
kwargs = {s[0] + '__sample_weight': sample_weight for s in model.steps}
model.fit(X, y, **kwargs)
Set parameter alpha to: original_alpha * np.sqrt(n_samples).
results
array([28.283, 23.142, 34.006, 28.283, 28.23 , 28.283, 13.798, 34.006,
18.863, 28.283, 13.798, 23.142, 17.272, 28.283, 23.142, 28.23 ,
23.142, 34.006, 34.006, 28.23 , 23.142, 34.006, 34.006, 13.798,
28.283, 23.142, 23.142, 18.863, 28.283, 28.283, 13.798, 18.863,
18.863, 34.006, 13.798, 34.006, 13.798, 28.283, 13.798, 13.798,
13.798, 28.283, 34.006, 28.283, 13.798, 13.798, 18.863, 34.006,
28.283, 28.283, 13.798, 23.142, 23.142, 28.23 , 18.863, 18.863,
18.863, 18.863, 23.142, 23.142, 13.798, 18.863, 28.23 , 23.142,
23.142, 28.283, 23.142, 28.283, 18.863, 13.798, 28.283, 13.798,
23.142, 23.142, 18.863, 28.283, 13.798, 13.798, 23.142, 18.863,
23.142, 23.142, 13.798, 34.006, 17.272, 34.006, 28.23 , 18.863,
18.863, 18.863, 28.23 , 34.006, 34.006, 28.23 , 18.863, 23.142,
23.142, 13.798])
testing_target
79 26.0
276 21.6
248 36.1
56 26.0
393 27.0
...
370 31.0
252 19.2
211 16.5
76 18.0
105 13.0
Name: MPG, Length: 98, dtype: float64
pd.DataFrame(list(results))
| 0 | |
|---|---|
| 0 | 28.283333 |
| 1 | 23.142222 |
| 2 | 34.005970 |
| 3 | 28.283333 |
| 4 | 28.230435 |
| ... | ... |
| 93 | 28.230435 |
| 94 | 18.862745 |
| 95 | 23.142222 |
| 96 | 23.142222 |
| 97 | 13.798246 |
98 rows × 1 columns
results2 = pd.DataFrame(list(results))
testing_target2 = pd.DataFrame(testing_target)
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error
y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]
print('rsquare: ',r2_score(results, testing_target))
print('meanabsoluteError: ',mean_absolute_error(testing_target, results))
test_results['ExtraTree']=mean_absolute_error(testing_target, results)
rsquare: 0.879077713080702 meanabsoluteError: 1.8407450367071052
Last week, we introduced model explainability with SHAP and will continue to incorporate it as part our model output this week. You can use the Kernel Explainer for explainability of both the Neural Networks and the TPOT classifier.
'''
copied from Explainer
#explainer = shap.KernelExplainer(f, X.iloc[:50,:])
#shap_values = explainer.shap_values(X.iloc[299,:], nsamples=500)
#shap.force_plot(explainer.expected_value, shap_values, X_display.iloc[299,:])
'''
import shap
explainer = shap.KernelExplainer(lambda x: dnn_model.predict(x).flatten(),train_features.iloc[:50,:])
shap_values = explainer.shap_values(test_features)
shap.initjs()
shap.force_plot(explainer.expected_value, np.array(shap_values), test_features)
# shows all explanations
2/2 [==============================] - 0s 3ms/step
1/1 [==============================] - 0s 32ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 2s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 30ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 2s 2ms/step 1/1 [==============================] - 0s 31ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 30ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 37ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 37ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 25ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 25ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 30ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 25ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 35ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 30ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 29ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 30ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 28ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 2ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 27ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 32ms/step 797/797 [==============================] - 1s 1ms/step 1/1 [==============================] - 0s 26ms/step 797/797 [==============================] - 1s 1ms/step
shap.initjs()
shap.force_plot(explainer.expected_value, np.array(shap_values), test_features)
explainer = shap.KernelExplainer(lambda x: dnn_model.predict(x).flatten(),train_features.iloc[:50,:])
shap_values = explainer.shap_values(test_features.iloc[0,:])
shap.initjs()
shap.force_plot(explainer.expected_value, np.array(shap_values), test_features.iloc[0,:])
# shows the first explanation
2/2 [==============================] - 0s 3ms/step 1/1 [==============================] - 0s 36ms/step 797/797 [==============================] - 2s 2ms/step
Let's take our models and make a model comparison demo like we did last week, but this time you're taking the lead!
st.DataFrame).test_results
{'linear_model': 2.4612009525299072,
'dnn_model': 1.6599880456924438,
'reloaded': 1.6599880456924438,
'dnn_model2': 2.289334297180176,
'ExtraTree': 1.8407450367071052}
!ls
dnn_model imports.ipynb dnn_model2 mlruns fuel-efficiency-prediction-streamlit.py requirements.txt fuel_efficiency.ipynb tpot_mpg_pipeline.py fuel_efficiency_sunday.ipynb
cat fuel-efficiency-prediction-streamlit.py
import pandas as pd
import plotly.express as px
import plotly.figure_factory as ff
import streamlit as st
import streamlit.components.v1 as components
from PIL import Image
# Add and resize an image to the top of the app
img_fuel = Image.open("../img/fuel_efficiency.png")
st.image(img_fuel, width=700)
st.markdown("<h1 style='text-align: center; color: black;'>Fuel Efficiency</h1>", unsafe_allow_html=True)
# Import train dataset to DataFrame
train_df = pd.read_csv("../dat/train.csv.gz", compression="gzip")
model_results_df = pd.read_csv("../dat/model_results.csv")
# Create sidebar for user selection
with st.sidebar:
# Add FB logo
st.image("https://user-images.githubusercontent.com/37101144/161836199-fdb0219d-0361-4988-bf26-48b0fad160a3.png" )
# Available models for selection
# YOUR CODE GOES HERE!
models = ["DNN", "TPOT"]
# Add model select boxes
model1_select = st.selectbox(
"Choose Model 1:",
(models)
)
# Remove selected model 1 from model list
# App refreshes with every selection change.
models.remove(model1_select)
model2_select = st.selectbox(
"Choose Model 2:",
(models)
)
# Create tabs for separation of tasks
tab1, tab2, tab3 = st.tabs(["🗃 Data", "🔎 Model Results", "🤓 Model Explainability"])
with tab1:
# Data Section Header
st.header("Raw Data")
# Display first 100 samples of the dateframe
st.dataframe(train_df.head(100))
st.header("Correlations")
# Heatmap
corr = train_df.corr()
fig = px.imshow(corr)
st.write(fig)
with tab2:
# YOUR CODE GOES HERE!
# Columns for side-by-side model comparison
col1, col2 = st.columns(2)
# Build the confusion matrix for the first model.
with col1:
st.header(model1_select)
# YOUR CODE GOES HERE!
# Build confusion matrix for second model
with col2:
st.header(model2_select)
# YOUR CODE GOES HERE!
with tab3:
# YOUR CODE GOES HERE!
# Use columns to separate visualizations for models
# Include plots for local and global explanability!
st.header(model1_select)
st.header(model2_select)
test_results
{'linear_model': 2.4612009525299072,
'dnn_model': 1.6599880456924438,
'reloaded': 1.6599880456924438,
'dnn_model2': 2.289334297180176,
'ExtraTree': 1.8407450367071052}
import csv
dataset.to_csv("../dat/train.csv")
dfresults = pd.DataFrame(test_results.items(), columns=['Model', 'Mean Absolute Error'])
dfresults.to_csv("../dat/model_results.csv")
!python fuel-efficiency-prediction-streamlit.py
Traceback (most recent call last):
File "/Users/Gilles/Documents/GitHub/week-07-intro-dl/nb/fuel-efficiency-prediction-streamlit.py", line 2, in <module>
import plotly.express as px
ModuleNotFoundError: No module named 'plotly'
!pip install plotly
Collecting plotly Using cached plotly-5.13.0-py2.py3-none-any.whl (15.2 MB) Collecting tenacity>=6.2.0 Using cached tenacity-8.1.0-py3-none-any.whl (23 kB) Installing collected packages: tenacity, plotly Successfully installed plotly-5.13.0 tenacity-8.1.0
!pip install streamlit
Collecting streamlit
Downloading streamlit-1.12.0-py2.py3-none-any.whl (9.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/9.1 MB 5.6 MB/s eta 0:00:0000:0100:01
Collecting pydeck>=0.1.dev5
Using cached pydeck-0.8.0-py2.py3-none-any.whl (4.7 MB)
Collecting pympler>=0.9
Using cached Pympler-1.0.1-py3-none-any.whl (164 kB)
Collecting validators>=0.2
Using cached validators-0.20.0.tar.gz (30 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: tornado>=5.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (6.2)
Requirement already satisfied: python-dateutil in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (2.8.2)
Requirement already satisfied: toml in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (0.10.2)
Collecting cachetools>=4.0
Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Requirement already satisfied: packaging>=14.1 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (22.0)
Requirement already satisfied: pandas>=0.21.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (1.4.4)
Requirement already satisfied: pyarrow>=4.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (10.0.1)
Collecting tzlocal>=1.1
Using cached tzlocal-4.2-py3-none-any.whl (19 kB)
Collecting semver
Using cached semver-2.13.0-py2.py3-none-any.whl (12 kB)
Collecting blinker>=1.0.0
Using cached blinker-1.5-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: gitpython!=3.1.19 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (3.1.30)
Collecting rich>=10.11.0
Downloading rich-13.3.1-py3-none-any.whl (239 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.0/239.0 kB 3.9 MB/s eta 0:00:0000:01
Requirement already satisfied: numpy in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (1.22.4)
Requirement already satisfied: requests>=2.4 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (2.28.1)
Collecting protobuf<4,>=3.12
Downloading protobuf-3.20.3-cp39-cp39-macosx_10_9_x86_64.whl (982 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 982.8/982.8 kB 4.4 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: typing-extensions>=3.10.0.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (4.4.0)
Requirement already satisfied: pillow>=6.2.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (9.3.0)
Collecting altair>=3.2.0
Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Requirement already satisfied: importlib-metadata>=1.4 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (4.11.3)
Requirement already satisfied: click>=7.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from streamlit) (8.0.4)
Requirement already satisfied: entrypoints in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from altair>=3.2.0->streamlit) (0.4)
Requirement already satisfied: toolz in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from altair>=3.2.0->streamlit) (0.12.0)
Requirement already satisfied: jsonschema>=3.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from altair>=3.2.0->streamlit) (4.16.0)
Requirement already satisfied: jinja2 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from altair>=3.2.0->streamlit) (2.11.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from gitpython!=3.1.19->streamlit) (4.0.10)
Requirement already satisfied: zipp>=0.5 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from importlib-metadata>=1.4->streamlit) (3.11.0)
Requirement already satisfied: pytz>=2020.1 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from pandas>=0.21.0->streamlit) (2022.7)
Requirement already satisfied: six>=1.5 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from python-dateutil->streamlit) (1.16.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from requests>=2.4->streamlit) (1.26.14)
Requirement already satisfied: certifi>=2017.4.17 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from requests>=2.4->streamlit) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from requests>=2.4->streamlit) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from requests>=2.4->streamlit) (2.0.4)
Collecting pygments<3.0.0,>=2.14.0
Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting markdown-it-py<3.0.0,>=2.1.0
Using cached markdown_it_py-2.1.0-py3-none-any.whl (84 kB)
Collecting pytz-deprecation-shim
Using cached pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: decorator>=3.4.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from validators>=0.2->streamlit) (5.1.1)
Requirement already satisfied: smmap<6,>=3.0.1 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from gitdb<5,>=4.0.1->gitpython!=3.1.19->streamlit) (5.0.0)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from jinja2->altair>=3.2.0->streamlit) (2.0.1)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from jsonschema>=3.0->altair>=3.2.0->streamlit) (0.18.0)
Requirement already satisfied: attrs>=17.4.0 in /Users/Gilles/anaconda3/lib/python3.9/site-packages (from jsonschema>=3.0->altair>=3.2.0->streamlit) (22.1.0)
Collecting mdurl~=0.1
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting tzdata
Using cached tzdata-2022.7-py2.py3-none-any.whl (340 kB)
Building wheels for collected packages: validators
Building wheel for validators (setup.py) ... done
Created wheel for validators: filename=validators-0.20.0-py3-none-any.whl size=19579 sha256=ce7e30f39e102f303a250163e0c9080775bd2ad370771554f848733ceaa2b2a9
Stored in directory: /Users/Gilles/Library/Caches/pip/wheels/5c/6a/fd/17a2b6f8d376e336d46414eeaf656327fd973158159b059046
Successfully built validators
Installing collected packages: validators, tzdata, semver, pympler, pygments, protobuf, mdurl, cachetools, blinker, pytz-deprecation-shim, pydeck, markdown-it-py, tzlocal, rich, altair, streamlit
Attempting uninstall: pygments
Found existing installation: Pygments 2.11.2
Uninstalling Pygments-2.11.2:
Successfully uninstalled Pygments-2.11.2
Attempting uninstall: protobuf
Found existing installation: protobuf 4.21.12
Uninstalling protobuf-4.21.12:
Successfully uninstalled protobuf-4.21.12
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
spyder 5.1.5 requires pyqt5<5.13, which is not installed.
spyder 5.1.5 requires pyqtwebengine<5.13, which is not installed.
Successfully installed altair-4.2.2 blinker-1.5 cachetools-5.3.0 markdown-it-py-2.1.0 mdurl-0.1.2 protobuf-3.20.3 pydeck-0.8.0 pygments-2.14.0 pympler-1.0.1 pytz-deprecation-shim-0.1.0.post0 rich-13.3.1 semver-2.13.0 streamlit-1.12.0 tzdata-2022.7 tzlocal-4.2 validators-0.20.0
Tensorflow playground for an interactive experience to understand how nueral networkds work.
An Introduction to Deep Learning for Tabular Data covers embeddings for categorical variables.
Imbalanced classification: credit card fraud detection demonstrates using class_weight to handle imbalanced classification problems.
This notebook is adapted from tensorflow/keras tuorial - regression
@title Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
@title MIT License
Copyright (c) 2017 François Chollet
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.